Natural Language Processing to Quantify Security Effort in the Software Development Lifecycle

Addressing security in the software development lifecycle is an ever-present concern for software engineers and organizations. From a management and monitoring perspective, it is difficult to measure 1) the amount of effort being focused on security concerns during active development and 2) the success of security related design and development efforts. Such data is simply not recorded. If reliable measurements were available, software project leaders would have a powerful tool to assess risk and inform decision making. This would enable managers to direct development and testing to assure a desired level of security in their software products, to protect both their organizations and customers. To fill this need and provide such data, we propose a technique for performing topic detection on data commonly available in most software development projects: text artifacts from issue tracking and version control systems. We apply machine learning and natural language processing techniques to create classifiers capable of accurately detecting whether a given text snippet is related to the topic of security. Realization of such a capability will give software teams the ability to analyze current and past levels of security effort, revealing immediate project focus and the long-term impacts of security tasking. We validate our approach via experiments on data from the large-scale open source Chromium software project. Our results show that a Naïve Bayes classification scheme using an n-gram feature-space is an appropriate and effective approach to automated topic detection of software security text snippets, and that effective training data can be derived from public data sources without the need for manual intervention. Keywords-natural language processing; machine learning; software security; security; topic defection; classification; naïve bayes